Reference Class Forecasting

Reference Class Forecasting, also called the Outside View, is a rule for making predictions by beginning with the outcomes of comparable cases rather than with the vivid details of the case at hand. Its decisive significance is corrective: it exposes how easily planners mistake an attractive internal story for a realistic forecast, and it supplies a disciplined way to confront the planning fallacy, base-rate neglect, unknown unknowns, and unjustified confidence.

Definition and Rationale

The Outside View asks what usually happens in a properly chosen class of similar cases. A baseline prediction is the estimate made when the forecaster knows only the category to which a case belongs. Specific information about the individual case may then justify an adjustment, but it should modify the baseline rather than replace it. This order matters because people who possess detailed information about one case rarely feel any need to seek statistics about the wider class; personal impressions therefore defeat distributional evidence before the comparison has even begun.

The rule was illustrated by a curriculum project developed for the Israeli Ministry of Education. After a team had spent about a year meeting every Friday afternoon, drafting chapters, and testing lessons, its members privately estimated how long it would take to submit a finished textbook. Their forecasts were based on the progress visible from inside the project. A separate question about similar teams produced a far darker reference class: 40 percent had failed, while successful projects had taken seven to ten years. The comparison suggested that the team’s outcome was slightly worse than that already grim baseline, even though the team’s internal estimates had seemed plausible.

The Planning Fallacy

The project exposed the planning fallacy, a term coined by Daniel Kahneman and Amos Tversky for forecasts that are unrealistically close to best-case scenarios. The team extrapolated from its earliest chapters and from its period of strongest commitment, treating the information immediately available as if it were the whole relevant evidence. That internal view omitted divorces, illnesses, bureaucratic coordination crises, and the long stretches in which no progress occurred. Such obstacles were not foreseeable in their particular form, yet the general likelihood that a large project would encounter serious obstacles was already visible in the reference class.

This distinction separates ignorance of a specific event from ignorance of the general risk of disruption. The team could not have named the future illnesses or divorces, but it could have recognized that large projects contain many possible routes to delay or failure. The same blindness affected other teams: they had apparently believed their projects feasible without knowing the odds of spending seven years on them or never completing them. The Outside View does not predict each obstacle; it estimates the consequences of belonging to a class in which obstacles are common.

The rule also changes who bears responsibility for optimistic plans. Unrealistic budgets and schedules may be produced partly because their authors want approval from superiors or clients, knowing that projects are seldom abandoned merely because costs and completion times exceed the original promises. Decision makers who approve such plans therefore commit the planning fallacy when they fail to demand an external comparison. Organizations can reduce the incentive by rewarding precise execution and penalizing failures to anticipate both foreseeable difficulties and the broader possibility of unknown unknowns.

Method of Application

Reference Class Forecasting has a practical three-step structure. First, identify an appropriate reference class, such as kitchen renovations or large railway projects. The class must be similar enough to illuminate the relevant risks rather than merely providing convenient numbers. Second, obtain the class statistics and use them to generate a baseline prediction. Relevant measures may include completion time, cost per mile, the percentage by which expenditures exceeded budget, failure rates, or the underperformance of projects of a particular type.

Third, use specific information about the individual case to adjust the baseline only when there is a clear reason to expect the project’s bias or outcome to differ from the class. The case-specific evidence is not discarded; it is placed in its proper role. The baseline anchors the forecast, while distinctive features move it away from the average only when those features have demonstrated predictive relevance. This procedure prevents a persuasive narrative about exceptional talent, commitment, or favorable circumstances from silently becoming a best-case assumption.

Bent Flyvbjerg’s implementation for transportation projects gives the method an institutional form. A large database containing plans and outcomes from hundreds of projects around the world can show likely cost and time overruns for different project categories. Authorities can use those distributions before approving public projects, and can estimate the budget reserve needed for overruns. Yet reserves can themselves invite expenditure: one official compared a budget reserve for contractors to red meat for lions. The method therefore supports realism without pretending that a contingency fund eliminates the forces that create overruns.

Evidence Against Intuitive Forecasting

The Outside View belongs to a wider challenge to the illusion of validity. Philip Tetlock’s twenty-year study gathered more than 80,000 predictions from 284 people who earned their living commenting on political and economic trends. The experts assessed events such as coups, wars, and emerging markets, assigning probabilities to persistence of the status quo, an increase, or a decrease. They performed worse than a simple procedure that assigned equal probabilities to the three outcomes, and specialists were not significantly better than nonspecialists even in regions they knew best.

Knowledge produced only a slight improvement, while the most knowledgeable forecasters were often less reliable because knowledge strengthened their illusion of skill and their overconfidence. Famous forecasters made more flamboyant predictions and were more overconfident than colleagues outside the limelight. The lesson is not that information has no value, but that information about a case can become a coherent story whose confidence exceeds its evidential power. Reference classes preserve information that an internal narrative tends to ignore.

The same pattern appears in financial forecasting. A Duke University survey collected 11,600 forecasts from chief financial officers of large corporations about the following year’s Standard and Poor’s index. The correlation between their estimates and the actual value was slightly less than zero, yet the forecasters did not appear to recognize that their predictions were worthless. Their stated 80 percent confidence intervals contained far too many surprises: 67 percent of outcomes fell outside the intervals, more than three times the expected 20 percent. Forecasting from what comes to mind makes it impossible to allow for information that never entered the story.

Calibration research reinforces the warning. A judge who is properly calibrated should find the assessed values falling below a stated percentile at the corresponding rate, with true values lying inside a 98 percent interval between the first and ninety-ninth percentiles on 98 percent of problems. In most studies, however, actual values fell outside the stated extreme bounds on about 30 percent of problems. Both naive and sophisticated judges produced intervals that were too narrow, and proper scoring rules did not eliminate the effect. Anchoring on a best estimate and making insufficient upward and downward adjustments helps explain why confidence distributions become tighter than the available knowledge warrants.

Broader Significance

Reference Class Forecasting is more than a project-management trick. It expresses a general discipline of moving from an individual case to the distribution that contains it, then returning to the case with appropriately limited adjustments. The approach is especially valuable when outcomes depend on luck, as in the contrast between a golfer’s first-day score and the score expected on the second day. An excellent score reflects both talent and unusually good luck; a poor score reflects both weakness and bad luck. Because the next day’s luck is unpredictable, the best forecast regresses toward average performance rather than repeating the extreme result.

That logic also clarifies why apparent success or failure can mislead judgments about skill. The flight instructors of the Israeli Air Force believed that praise caused cadets to perform worse and punishment caused them to improve, because performance commonly deteriorated after an unusually good attempt and improved after an unusually bad one. The change could be regression to the mean rather than an effect of reward or punishment. A reference-class perspective similarly prevents a planner from treating a single promising trajectory as proof that the project is exceptional.

The rule is therefore a restraint on narrative confidence. It does not deny that cases differ, nor does it demand mechanical obedience to an average. It insists that differences be demonstrated rather than imagined, that the distributional evidence be made visible before a decision is approved, and that forecasts be judged against outcomes rather than against the eloquence of the story used to produce them. In environments where plans are complex, feedback is delayed, and failure has many unforeseeable forms, the Outside View provides the most reliable way to locate a prediction in the real world.